ASRS-CMFS vs. RoBERTa: Comparing Two Pre-Trained Language Models to Predict Anomalies in Aviation Occurrence Reports with a Low Volume of In-Domain Data Available

We consider the problem of solving Natural Language Understanding (NLU) tasks characterized by domain-specific data. An effective approach consists pre-training Transformer-based language models from scratch using data before fine-tuning them on task at hand. A low volume is problematic in this context, given that performance relies heavily abundance during pre-training. To study problem, we create a benchmark replicating realistic field use to classify aviation occurrences extracted Aviation Safety Reporting System (ASRS) corpus. compare two new benchmark: ASRS-CMFS, compact model inspired RoBERTa, pre-trained only little data, and regular RoBERTa model, with no The benefits its size advantage, while ASRS-CMFS strategy. find compelling statistical evidence outperforms but show more compute-efficient than RoBERTa. suggest good strategy for NLU context scarcity.